Introduction to Machine Learning

Chapter 04: Model Evaluation, Generalization and Hyperparameter Tuning

1. Introduction

Evaluating a machine learning model on the data used to train it reveals very little about how well it will perform in practice. Just as a student can score 100% on an exam by memorizing the answer key, a model can achieve near-zero training error simply by memorizing its input data. This chapter focuses on the methodologies and tools that separate a model that has genuinely learned underlying patterns from one that has merely memorized noise.

We begin by distinguishing between internal model parameters and user-configured hyperparameters. Next, we build up formal evaluation protocols—moving from simple hold-out and three-way splits to bias–variance diagnostics, learning curves, and k-fold cross-validation. Finally, we connect these concepts to practical scikit-learn workflows, showing how to execute systematic hyperparameter tuning without contaminating your final evaluation.

Learning Objectives

2. Theory

Reliable model evaluation begins by separating what the model learns from what the practitioner chooses, then using disciplined validation procedures to understand generalization, diagnose fitting behavior, and select hyperparameters without contaminating the final test estimate.

2.1 Parameters vs. Hyperparameters

A model has two different kinds of settings, and it is important to keep them apart. Some are found by the learning algorithm itself, while others must be fixed by the user before training begins. The table below contrasts the two.

Parameters
Hyperparameters

Learned automatically from training data during fitting.

  • Data-dependent; change when training data changes.
  • Internal to the model — define its structure once trained.
  • Examples:
AlgorithmParameters
Linear / Logistic RegressionCoefficient vector β and intercept β₀
Neural NetworkWeights W and biases b of every connection
Decision TreeActual split conditions and thresholds at each node
KNN(None — KNN stores all training points directly)

Set by the user before training starts. They control the learning process itself.

  • Not learned from data; stay fixed during training.
  • Require tuning — different values dramatically change performance.
  • External to the core model; guide how parameters are learned.
  • Examples:
AlgorithmHyperparameters
KNNk (neighbors), weights (uniform/distance), metric, p (Minkowski)
Neural NetworkLearning rate α, #layers, #neurons/layer, batch size
Decision Treemax_depth, min_samples_leaf, splitting criterion (gini/entropy)
Ridge/Lasso RegressionRegularization strength α

2.2 Model Complexity, Generalization, and the KNN Complexity Ladder

Having distinguished parameters from hyperparameters, the central question becomes: how do our hyperparameter choices dictate performance? Hyperparameters directly govern model complexity—the flexibility of a model to fit patterns in the data. This complexity sits in direct tension with generalization, which is the model's ability to perform well on unseen data. k-Nearest Neighbors (k-NN) illustrates this trade-off clearly, as adjusting a single hyperparameter (k) moves the model across the entire complexity spectrum.

K value (KNN)# Effective ParametersModel ComplexityTypical Behavior
K = 10 (stores data)HighestMemorizes every training point
K = 30HighVery flexible, jagged boundaries
K = 100MediumModerately smooth
K = 1000LowSmooth, simple boundaries
K = n (all points)0LowestConstant majority-class baseline

2.3 Underfitting vs. Overfitting

Comparing performance across training and held-out data reveals whether a model has learned true underlying patterns or merely memorized noise. When model complexity is set too low or too high, performance degrades in predictable ways. The tabs below detail the two classic failure regimes—underfitting and overfitting—alongside the balanced state we aim to achieve.

Underfitting (Too Simple)
Overfitting (Too Complex)
Just Right
  • Symptoms: High training error, high validation error. Both curves are bad and close to each other.
  • Analogy: Using a straight line to fit a curved parabola — can't express the true relationship.
  • Diagnosis: High bias (systematic error; model makes the same wrong assumptions every time).
  • Fixes: Increase model complexity, add more features, decrease regularization, train longer (NN).
  • Symptoms: Very low training error, much higher validation error. Large gap between train and val curves.
  • Analogy: Memorizing every past exam's exact answer instead of understanding the concepts; fails on any new phrasing.
  • Diagnosis: High variance (predictions change wildly depending on which rows happen to land in the training set).
  • Fixes: Gather more data, decrease model complexity (increase k for KNN), add regularization, remove noisy features, apply early stopping (NN).
  • Symptoms: Training error and validation error are both acceptably low, with a small and stable gap. Validation error flattens and does not spike back upward.
  • Diagnosis: Low bias AND low variance — the sweet spot. The complexity setting where validation error is minimized is your hyperparameter target.
  • Learning curves by training set size (next section) help you distinguish "need more data" vs. "need different model".

2.4 The Bias-Variance Decomposition (Optional)

To understand the root causes of underfitting and overfitting, we turn to the bias–variance decomposition. This framework provides a formal mathematical foundation for generalization by breaking down a model's expected prediction error on unseen data into three distinct sources:

\( \boxed{ \text{Expected Error} = \text{Bias}^2 + \text{Variance} + \text{Irreducible Noise} } \)

Conceptual Breakdown: Systematic Error vs. Prediction Variance

Consider evaluating model performance across multiple independent training sets drawn from the same data source:

Under this framework:

Mathematical interpretation. The key mathematical detail is that \( \bar{\hat{y}} \) is computed separately for each input point: we hold \(x\) fixed and average the predictions obtained from many different training datasets, while the bias and variance themselves are evaluated at that fixed point.

To make this precise, suppose the observed target at a particular input \(x\) is generated by

\( y = f(x) + \varepsilon, \qquad E[\varepsilon]=0,\qquad \mathrm{Var}(\varepsilon)=\sigma^2. \)

Let \( \hat{y}_D=\hat{f}_D(x) \) be the prediction produced by a model trained on one particular dataset \(D\), and let

\( \bar{\hat{y}} = E_D[\hat{y}_D] \)

be its average prediction at the same \(x\) across repeated training datasets. The expected squared prediction error then decomposes as:

\( \boxed{ E_{D,\varepsilon}\!\left[(y-\hat{y}_D)^2\right] = \underbrace{(f(x)-\bar{\hat{y}})^2}_{\text{Bias}^2} + \underbrace{E_D\!\left[(\hat{y}_D-\bar{\hat{y}})^2\right]}_{\text{Variance}} + \underbrace{\sigma^2}_{\text{Irreducible Noise}} } \)

When we consider many test points, we repeat this decomposition for each \(x_i\) and then sum or average the resulting errors across those points.

A Useful Caveat: The True Function \(f(x)\) Is Usually Unknown

The decomposition compares predictions with the underlying, noise-free function \(f(x)\), not simply with one observed target \(y\). In real-world data, we usually observe only \(y=f(x)+\varepsilon\), so we usually cannot directly calculate the true Bias² and irreducible noise for a particular problem. The value of the decomposition in practice is therefore mainly diagnostic: it gives us a framework for understanding whether poor performance is mainly due to an overly simple model (high bias) or an overly sensitive model (high variance).

Worked Example: Bias and Variance at One Point

Suppose that for one test point \(x_0\), the true function value is \(f(x_0)=10\). We train the same model on three different training datasets, obtaining predictions:

\( \hat{y}_{D_1}=7,\qquad \hat{y}_{D_2}=9,\qquad \hat{y}_{D_3}=11. \)

Step 1 — Average prediction:

\( \bar{\hat{y}}=\frac{7+9+11}{3}=9. \)

Step 2 — Bias²: The average prediction is 9, while the true function value is 10.

\( \text{Bias}^2=(f(x_0)-\bar{\hat{y}})^2=(10-9)^2=1. \)

Step 3 — Variance: The individual predictions fluctuate around their average of 9.

\( \text{Variance} =\frac{(7-9)^2+(9-9)^2+(11-9)^2}{3} =\frac{4+0+4}{3} =\frac{8}{3}\approx2.67. \)

Interpretation: The model has some systematic error because its average prediction is below the truth, giving Bias² = 1. It also has prediction variability because different training datasets produce different predictions, giving Variance ≈ 2.67. A model with predictions that stay close to one another but consistently miss the true value would have high bias and low variance; a model whose average is close to the truth but whose predictions swing widely would have low bias and high variance. The same logic scales up: training on 100 datasets instead of 3 just gives a more reliable estimate of \(\bar{\hat{y}}\) and the spread around it.

Irreducible noise is different: even if the model has zero bias and zero variance, the observed target can still differ from \(f(x)\) because of randomness or measurement noise in the data-generating process. This is the part no learning algorithm can eliminate.

Because irreducible noise sets a hard lower bound on prediction error, model optimization is fundamentally a balancing act: finding the complexity at which the sum of Bias² and Variance is minimized. In practice, though, we cannot compute Bias² and Variance directly, since \(f(x)\) is unknown — that's the caveat noted above. The sections that follow — two-way and three-way splits, cross-validation, and learning curves — are the practical tools used to estimate whether a model's error is dominated by bias or by variance, so that its complexity can be adjusted accordingly.

2.5 Hold-Out Method — Basic 2-Way Split

To measure generalization empirically rather than relying on theoretical assumptions, we must establish a disciplined data-splitting protocol. The simplest and most intuitive approach is the hold-out method, which isolates a portion of our data solely for final evaluation.

Full labeled dataset split into training and test sets Eighty percent of the labeled dataset is used for training and twenty percent is held out untouched until final evaluation. FULL LABELED DATASET 80% FOR TRAINING 20% HELD OUT TRAINING SET Used for: Fitting the model Learning parameters (β, θ, splits, etc.) TEST SET Locked in a box Evaluated ONCE at the END RULE Test set stays TRULY untouched until the FINAL evaluation. s*

The model is fitted on the training set only, and the test set is held out until the end. The test set error estimates generalization error (out-of-sample error) — how well the model will perform on truly unseen data. Low training error + high test error = overfitting.

The Hold-Out Flaw for Hyperparameter Tuning

If we reuse the test set repeatedly for model selection or hyperparameter tuning, it effectively becomes part of the training data and the model overfits to the test set. The reported scores then become optimistic: they are inflated, misleading, and not reproducible on truly unseen data.

2.6 The Three-Way Split — Train / Validation / Test

A standard two-way split works well for evaluating a single, fixed model. However, when we repeatedly tune hyperparameters against the test set, information leaks from the test set into our design decisions. To prevent this data contamination, we expand our architecture into a three-way split.

Full Labeled Dataset Split A diagram showing training, validation, and held-out test datasets with the model selection and final evaluation workflow. FULL LABELED DATASET ~64% TRAIN ~16% VALIDATION 20% TEST (HELD OUT!) Train Validation Test • Fit model • Learn params • Rank hyperparams by CV score Evaluated exactly ONCE after best model is selected! RETRAIN BEST CONFIG ON TRAIN + VAL then evaluate Model fitting Hyperparameter selection Final unbiased evaluation s*
  1. Training set (~64%): Fit different models with many hyperparameter values.
  2. Validation set (~16%): Evaluate each trained model; pick whichever hyperparameters perform best.
  3. Retrain the winning configuration on TRAIN + VAL combined.
  4. Test set (20%): Evaluate the final retrained model exactly once — that number is your reported generalization.

2.7 K-Fold Cross-Validation

While a three-way split provides a clean separation of data, discarding 15–20% of a small dataset solely for validation can significantly degrade model quality. K-fold cross-validation resolves this trade-off by systematically rotating the validation set across the entire training cohort.

The hold-out validation estimates are sensitive to exactly which rows landed in the validation split. K-fold fixes this by repeating the process k times on different partitions and averaging:

K-fold cross-validation workflow Training and validation portion of the data, showing iterations where each fold is used once as a test set and the remaining folds are used for training. Training + Validation Portion K-fold cross-validation partitions the available data into reusable training and test folds. 80 rows shown as 80% Fold structure The 80% training + validation portion is divided into k folds. Fold 1 Fold 2 Fold 3 Fold 4 Fold 5 Fold 6 ⋯ Fold k Cross-validation iterations TEST fold TRAIN folds Fold allocation Validation score Iteration 1 TESTTRAINTRAINTRAINTRAINTRAIN⋯TRAIN score₁ Iteration 2 TRAINTESTTRAINTRAINTRAINTRAIN⋯TRAIN score₂ Iteration 3 TRAINTRAINTESTTRAINTRAINTRAIN⋯TRAIN score₃ ⋮ additional fold rotations ⋮ Iteration k TRAINTRAINTRAINTRAINTRAINTRAIN⋯TEST scoreₖ Final CV score Average the validation score from every fold rotation. average(score₁, score₂, score₃, …, scoreₖ) s*

K-Fold CV Steps

  1. Randomly partition the training + validation rows into k disjoint, equal-sized folds without replacement.
  2. For i = 1 … k: Train on folds 1..k except i; evaluate on fold i → get scoreᵢ.
  3. Report the mean of the k scores (and optionally their standard deviation) as the CV performance estimate.
  4. After hyperparameters are chosen via CV, retrain the final model on the entire train+val set.
  5. Do the final evaluation on the still-locked-away test set.

Standard value: k = 10 (10-fold CV) is the default in nearly every ML paper. Stratified k-fold (for classification) ensures each fold has roughly the same class distribution as the whole dataset.

2.8 Leave-One-Out Cross-Validation (LOOCV)

In standard K-fold cross-validation, data is split into a small number of equal folds (typically k = 5 or k = 10). However, a natural boundary case arises when we push k to its logical extreme: setting k = n, where n is the total number of instances in the dataset. This variation is known as Leave-One-Out Cross-Validation (LOOCV).

In LOOCV, each individual fold consists of a single observation. For every iteration, the model trains on n - 1 samples and is evaluated on the one remaining held-out row. This process repeats n times so that every single data point serves as the test set exactly once.

2.9 Learning Curves — Two X-Axis Families

While numerical evaluation scores tell us how well a model performs, learning curves reveal why it succeeds or fails. By tracking performance across changing sample sizes or complexity settings, learning curves operationalize the bias-variance framework into visual diagnostics.

Two complementary plotting habits diagnose different failure modes:

X = Training Set Size
X = Model Complexity

Plot train error (decreasing curve) and validation error (decreasing then plateau) against the number of training rows:

  • Large gap persists even with max data: High variance → need simpler model or more regularization, not more data.
  • Both curves converge high (at a bad error): High bias → need more complex model, more features.
  • Both curves still decreasing at the far right of the plot: Collect more training data — the model isn't saturated yet.

For KNN, vary k (small k = more complex) on the X axis against train + val accuracy on the Y axis.

Accuracy versus model complexity Training accuracy rises as model complexity increases, while validation accuracy rises to a maximum sweet spot and then falls. Accuracy vs. model complexity Training improves continuously; validation performance peaks at the right level of complexity. 1.0 0.5 0.0 Accuracy Training accuracy rises Validation accuracy Sweet spot Validation accuracy is maximized k = 1 k = 10 k = n complex simple Model complexity (k small → large) s*

Pick the complexity where validation set accuracy is maximal (or validation loss minimal).

2.10 sklearn cross_val_score and n_jobs

from sklearn.model_selection import cross_val_score # KNN with K=19, stratified 10-fold CV, accuracy scores = cross_val_score(model, X_trainval, y_trainval, cv=10, scoring='accuracy', n_jobs=-1) print("CV scores per fold:", scores) print("Mean CV accuracy: %.3f (SD %.3f)" % (scores.mean(), scores.std()))

n_jobs parallelizes across CPU cores: n_jobs = 1 → sequential; n_jobs = 2 → two folds at once on 2 CPUs; n_jobs = −1 → all available CPUs.

2.11 Grid Search for Hyperparameter Tuning

Grid search = brute-force exhaustive sweep over a user-specified Cartesian grid of hyperparameter combinations. For each combination, run K-Fold CV and record its mean CV score; then pick the combination with the best score.

from sklearn.model_selection import GridSearchCV from sklearn.neighbors import KNeighborsClassifier param_grid = { 'n_neighbors': [3, 5, 7, 11, 15, 19, 25, 31], 'weights': ['uniform', 'distance'], 'metric': ['euclidean', 'manhattan'], } gs = GridSearchCV(KNeighborsClassifier(), param_grid, cv=10, scoring='accuracy', n_jobs=-1) gs.fit(X_trainval, y_trainval) print("Best CV score %.3f" % gs.best_score_) print("Best params:", gs.best_params_) final_model = gs.best_estimator_ # pre-refit on full trainval print("Test accuracy %.3f" % final_model.score(X_test, y_test))

Learning curves vs. Grid search

3. Interactive Examples

3.1 Parameter or Hyperparameter?

Classify each item as a Parameter or a Hyperparameter.

(A) In a Decision Tree: the maximum depth restriction.

Hyperparameter. The user sets the maximum depth before training begins. The actual splits the tree learns are parameters, but the limit restricting them is a hyperparameter.

(B) In Linear/Logistic Regression: the learned coefficient β₁ (the slope).

Parameter. The coefficient β₁ is learned directly from the data during the training process. It is not set by the user. (Note: The regularization strength α that penalizes it would be a hyperparameter).

(C) In Neural Networks: the learning rate (α) of gradient descent.

Hyperparameter. The learning rate controls the step size during optimization. It is not learned from the data; you try several values and pick the best via cross-validation.

3.2 Train / Val / Test Split Decisions

Five mini-scenarios. For each, answer: is this allowed ML practice, or does it leak data / invalidate the test score?

  1. "After training, I use the test set accuracy to decide between k=3 and k=5 neighbors."
  2. "I use 10-fold CV on TRAIN+VAL to pick k, then retrain on the full train+val, and evaluate the final score on the held-out test."
  3. "I apply StandardScaler to the WHOLE dataset first, then split into train/test."
  4. "I split first, use StandardScaler().fit(X_train) only, then transform X_train and X_test."
  5. "I use cross_val_score with StandardScaler inside a Pipeline wrapping both scaling + KNN."
  1. ❌ Data Leakage! Picking k based on test set performance means you are indirectly fitting your model to the test set.
  2. ✅ Correct. This is the classic, strict 3-way split protocol.
  3. ❌ Data Leakage! Fitting the scaler on the whole dataset means the test set's mean/std pollute the training preprocessing.
  4. ✅ Correct. Fit on train only; transform both. This prevents leakage.
  5. ✅ Best Practice. A Pipeline + CV ensures that scaling is fit independently inside each CV training fold, preventing leakage at the fold level.

3.3 Manual 3-Fold CV by Hand

Six labeled 1-D points: X = [1, 2, 3, 4, 5, 6] and y = [A, B, A, B, A, B].

Fold 1 = {1,2}, Fold 2 = {3,4}, Fold 3 = {5,6}. Use 1-NN. Compute per-fold accuracy, then mean CV accuracy.

(i) Fold 1 Test: Train on {3,4,5,6}, Test on {1:A, 2:B}.
Nearest to X=1 among {3,4,5,6} is X=3 (A) → predict A (Correct).
Nearest to X=2 among {3,4,5,6} is X=3 (A) → predict A (Wrong).
Accuracy = 1/2 = 0.5

(ii) Fold 2 Test: Train on {1,2,5,6}, Test on {3:A, 4:B}.
Nearest to X=3 among {1,2,5,6} is X=2 (B) → predict B (Wrong).
Nearest to X=4 among {1,2,5,6} is X=5 (A) → predict A (Wrong).
Accuracy = 0/2 = 0.0

(iii) Fold 3 Test: Train on {1,2,3,4}, Test on {5:A, 6:B}.
Nearest to X=5 among {1,2,3,4} is X=4 (B) → predict B (Wrong).
Nearest to X=6 among {1,2,3,4} is X=4 (B) → predict B (Correct).
Accuracy = 1/2 = 0.5

Mean CV Accuracy = (0.5 + 0.0 + 0.5) / 3 ≈ 0.333.
Insight: A crucial detail here is that the nearest neighbor must come from the training fold only — never from the other test point in the same fold.

3.4 Diagnose the Learning Curve

Three learning-curve scenarios. Match each to its diagnosis and recommendation:

  1. Scenario A: Training accuracy 99%, validation accuracy 72%, large gap. Adding more training data doesn't shrink the gap.
  2. Scenario B: Training accuracy 68%, validation accuracy 66%, both low and close together. Adding more data barely helps.
  3. Scenario C: Training accuracy drifts from 99% (100 samples) to 90% (10,000 samples). Validation accuracy starts at 55%, rises monotonically, and is still climbing at 10,000 samples.
  1. High Variance / Overfitting. Recommendation: Make the model simpler (increase k for KNN, increase regularization, reduce feature count).
  2. High Bias / Underfitting. Recommendation: Make the model more complex (decrease k, add features, decrease regularization, try a more powerful model).
  3. Converging Curves — Need More Data. Both curves are still moving toward each other and the validation curve hasn't flattened out. Acquire more labeled rows.

4. Numerical Solutions

Problem 1: K-Fold on Small Dataset

Small n = 150 labeled training rows. Perform a stratified k = 5 stratified CV.

  1. How many rows are test in each fold?
  2. In each iteration, how many rows are used for training?
  3. How many distinct models are fitted in total?
  4. Suppose the fold accuracies are {0.87, 0.90, 0.83, 0.87, 0.93}. Report the 5-fold CV mean accuracy and its standard deviation (sample).
📘 Show Solution

(a) 150 / 5 = 30 rows per fold.

(b) 150 − 30 = 120 training rows per iteration.

(c) 5 folds → 5 separate model fits → 5 models. (Then +1 final refit on all 150 rows once hyperparameters are chosen, for a total of 6 fits.)

(d)

\( \bar{x} = \frac{0.87 + 0.90 + 0.83 + 0.87 + 0.93}{5} = \frac{4.40}{5} = \mathbf{0.88} \)
\( \text{Sample SD} = \sqrt{\frac{(−0.01)^2 + (0.02)^2 + (−0.05)^2 + (−0.01)^2 + (0.05)^2}{4}} = \sqrt{0.0014} \approx \mathbf{0.0374} \)

Report: CV accuracy = 88.0% (± 3.7%).

Problem 2: 10-Fold vs. LOOCV on n = 81 Samples

  1. How many total model fits does 10-fold require?
  2. How many for LOOCV?
  3. Give one statistical advantage of 10-fold: (i) n = 1000 labeled dataset. (ii) n = 15 dataset?
📘 Show Solution

(a) 10 → 10 fits (plus 1 final refit = 11).

(b) LOOCV = n = 81 rows → 81 fits (plus 1 refit → 82 total).

(c) (i) n = 1000: 10-fold clearly better — 10 models instead of 1000, plus each fold has ~900 training rows which is plenty; LOOCV would be overkill. (ii) n = 15: 10-fold leaves only 1–2 test samples per fold — scores unreliable. LOOCV trains on 14 rows, tests 1, no randomness → better estimate for tiny datasets; prefer LOOCV!

Problem 3: KNN Complexity Curve by Hand

On a small 2-D toy binary problem, you test KNN with k = 1, 3, 7, 15 and measure both training accuracy and 5-fold CV (validation) accuracy: {k, train, CV} triples are {1, 1.00, 0.62}, {3, 0.95, 0.78}, {7, 0.88, 0.85}, {15, 0.78, 0.77}.

  1. Identify which k values show symptoms of overfitting, underfitting, and "just right".
  2. Which k should you pick for deployment? Why?
  3. Sketch the qualitative train and CV curves on scratch paper and confirm the "inverted U" CV shape is present.
📘 Show Solution

(a) k = 1: train accuracy 100% (memorized) — CV only 62% with big gap → classic overfitting / high variance. k = 15: both errors are fairly high but close together → underfitting / high bias (too smooth, ignoring local structure). k = 3 & k = 7: moving toward just right as k rises to 7.

(b) Pick k = 7. It has the maximum cross-validation accuracy = 85%, with train (88%) and CV (85%) only 3 pp apart → low gap, low overfit.

(c) CV accuracy: k = 1 → 0.62, k = 3 → 0.78, k = 7 → 0.85 (peak!), k = 15 → 0.77 (falling back). That inverted-U shape is the complexity curve in action.

Problem 4: Grid Search Combinatorics

GridSearchCV with param_grid = { n_neighbors: [5, 11, 19, 27, 35], weights: ['uniform', 'distance'], metric: ['euclidean', 'manhattan', 'chebyshev'] }. Stratified 5-fold CV.

  1. How many distinct hyperparameter combinations?
  2. Total distinct classifier fits (not counting final refit)?
  3. If a single KNN fit takes 0.2 seconds on this dataset, roughly what wall-clock runtime with n_jobs = -1 on an 8-CPU machine?
📘 Show Solution

(a) Cartesian product: 5 k values × 2 weights × 3 metrics = 30 combinations.

(b) Each combination has 5 CV folds → 5 fits. 30 × 5 = 150 fits (plus 1 final refit on winner → 151 total).

(c) Sequential: 150 × 0.2 s = 30 s. With 8 CPUs in parallel: ~30/8 ≈ 3.75 seconds (plus small overhead — very fast!). This is one of grid search's advantages — it's embarrassingly parallel.

5. Try It Yourself

Problem 1: 3-Way Split Sizes

Dataset of 2,500 rows. Use 64/16/20 train/val/test split.

  1. How many samples land in each split?
  2. Which split is used to pick k?
  3. After k is picked, which split(s) do you retrain the final model on?
  4. Which split do you evaluate the final retrained model on, and how many times?
📘 Show Solution

(a) 2500 × 0.64 = 1,600 train; × 0.16 = 400 val; × 0.20 = 500 test.

(b) The validation set — or via k-fold on the combined train+val (2,000 rows).

(c) Train + Validation combined (2,000 rows).

(d) Evaluate exactly once on the 500-row test set. One single number — that is the reported generalization accuracy.

Problem 2: K-Fold CV with sklearn Workflow

You compare four KNN classifiers on a binary classification task: k ∈ {3, 7, 15, 31}. 5-fold CV gives fold accuracies below:

kFold1Fold2Fold3Fold4Fold5
30.850.820.880.800.85
70.890.860.900.870.88
150.880.890.860.910.91
310.820.830.810.840.85
  1. Calculate mean CV accuracy per k and pick the best k.
  2. Compute sample SD of CV scores for both k = 7 and k = 15. Which is more stable?
  3. After picking the best k, describe in 1–2 sentences what you do next.
📘 Show Solution

(a) Means:

  • k = 3: 4.20 / 5 = 0.840
  • k = 7: 4.40 / 5 = 0.880
  • k = 15: 4.45 / 5 = 0.890 ← best mean
  • k = 31: 4.15 / 5 = 0.830

(b) SD(k=7): values around 0.880 → devs [+0.01, −0.02, +0.02, −0.01, 0.00] → var = 0.00025 → SD = 0.0158. SD(k=15): mean = 0.890 → devs [−0.01, 0, −0.03, +0.02, +0.02] → var = 0.00045 → SD ≈ 0.0212. k = 7 slightly more stable; both good. Winner k = 15 wins by mean accuracy.

(c) Retrain a single KNN(k=15) classifier on the full TRAIN+VAL combined dataset, then evaluate exactly once on the held-out test set. Report that single value as your generalization accuracy.

Problem 3: LOOCV on Tiny n = 4

4 points: X = [1, 2, 4, 5]; y = [A, A, B, B]. 1-NN classifier. Compute LOOCV accuracy.

📘 Show Solution

Iter 1: test 1 (A). Train on {2(A), 4(B), 5(B)}. Nearest of 1 is 2(A) → predict A correct ✔

Iter 2: test 2 (A). Train {1(A), 4(B), 5(B)}. Nearest is 1(A) → predict A correct ✔

Iter 3: test 4 (B). Train {1(A), 2(A), 5(B)}. Nearest is 5(B) → predict B correct ✔

Iter 4: test 5 (B). Train {1(A), 2(A), 4(B)}. Nearest is 4(B) → predict B correct ✔

LOOCV accuracy = 4 / 4 = 1.00 (100%).

Problem 4: Learning Curve Prescription

A neural network gives training loss 0.001, validation loss 0.65. Your colleague suggests: "We just need more labeled data." Critique that suggestion by (a) naming the actual syndrome, then (b) giving three concrete interventions that address it directly, and (c) identifying one diagnostic observation on the curve that would actually justify "get more data."

📘 Show Solution

(a) Classic high-variance / overfitting (huge train/val gap). (b) Three fixes from the menu: (i) simplify architecture (fewer layers/neurons), (ii) add dropout or weight regularization, (iii) add data augmentation / noise, (iv) apply early stopping, (v) feature selection to remove noisy inputs, (vi) decrease model complexity (e.g., bigger k if it were KNN). (c) "Need more data" is justified only when the validation loss curve is still decreasing at the right edge of the training-set-size X-axis and not yet plateaued. If it's flat with a big gap, more rows won't close it — the model is too flexible.

Problem 5: Grid Search with a Pipeline

You want to compare KNN hyperparameters but also need to standardize features. Why is Pipeline([('sc', StandardScaler()), ('clf', KNeighborsClassifier())]) required inside GridSearchCV instead of scaling once at the top level? Give the one-sentence leakage explanation, then write the param_grid format with pipeline namespaced keys.

📘 Show Solution

Leakage explanation: Scaling before CV means each fold's StandardScaler was fit using test-fold rows as part of its mean/SD — the validation fold's distribution statistics leak into training, producing optimistically biased CV scores. The pipeline re-fits scaler + classifier on each fold's training split only, so CV is honest.

Namespaced grid format:

param_grid = { 'clf__n_neighbors': [3, 7, 11, 15, 21], 'clf__weights': ['uniform', 'distance'], 'clf__metric': ['euclidean', 'manhattan'], }

6. Interactive Quiz

Answer all 7 questions. Click an option for instant feedback.

Your score: 0 / 7

7. Key Takeaways

  1. Parameters are learned; hyperparameters are set. Don't confuse them — parameters come from data; the other requires tuning via CV.
  2. Three-way split: Train / Validation / Test (64/16/20). Never pick hyperparameters on val; never report final performance on test.
  3. Never tune hyperparameters on the test set. It leaks data; gives optimistic scores. Use validation (or CV on trainval).
  4. K-fold CV gives a less-biased, lower variance estimate. Repeat hold-out across k folds, average the k fold scores. k=10 is the standard default.
  5. Stratified k-fold. Use for classification to preserve each fold's class distribution.
  6. LOOCV = k = n. Use only for very small datasets; expensive but deterministic.
  7. Retrain winner on all trainval after picking via CV. Then evaluate once on test. That single number is your paper-ready result.
  8. Underfitting → high bias; overfitting → high variance. Use train-vs-val learning curves (both size-X and complexity-X) to diagnose which disease you have.
  9. Validation-accuracy peak = Optimum complexity. For KNN: find the k where CV score is maximized. Increase k → simpler model (less overfit, more underfit). Decrease k → opposite.
  10. When both curves are bad & close: need more complex model / more features. When they are far apart: need simpler model / more regularization / better feature selection. When both still climbing at max data: get more labeled rows.
  11. F1 uses the harmonic mean, not arithmetic mean, exactly to prevent gaming one metric while failing the other. Harmonic mean ≤ arithmetic and always closer to the worse of the two values.

8. Common Pitfalls

  1. Test-set reuse = data leakage. Each time you peek at test accuracy and change the model, test information flows backward. Stop — use CV on the trainval set instead.
  2. Scaling before splitting. StandardScaler fit on whole dataset uses test-set mean/std. Fix: split → fit(train) → transform(train), transform(test).
  3. No Pipeline inside CV. If you scale the entire dataset once outside cross_val_score, each fold's training portion sees the validation fold's scaling statistics. Fix: use a Pipeline(StandardScaler → KNN) so each CV fold learns its own scaler from only that fold's training rows.
  4. Choosing k = 2 (binary ties). Use odd k to avoid 50/50 ties.
  5. LOOCV on n = 10 000. 10 000 fits would take days. 10-fold is essentially as accurate and 1000× cheaper.
  6. Reporting CV accuracy as the final number. CV chooses the hyperparameters. Final number is test accuracy after retrain on full trainval on that winner.
  7. Grid search with pre-scaled data. Scaler fits leak test-fold information. Put scaler + classifier in a Pipeline inside GridSearchCV.